Model Checking



In [1]:

    
%load_ext autoreload
%autoreload 2
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd

from load_utils import *



In [2]:

    
d = load_diffs(keep_diff = True)
df_events, df_blocked_user_text = load_block_events_and_users()

Q: How does the distribution over attack probabilities compare across annotators vs models?



In [3]:

    
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure(figsize=(12, 6))

xa = np.sort(d['annotated']['recipient_score'])
ya = 1. * np.arange(len(xa)) / (len(xa) - 1)

xm = np.sort(d['annotated']['pred_recipient_score'])
ym = 1. * np.arange(len(xm)) / (len(xm) - 1)

ax1 = fig.add_subplot(121)
ax1.plot(xa, ya, label = 'annotators')
ax1.plot(xm, ym, label = 'model')
ax1.set_xlabel('$P(X<x)$')
ax1.set_ylabel('$p$')
ax1.legend()


ax2 = fig.add_subplot(122)
ax2.plot(xa, ya, label = 'annotators')
ax2.plot(xm, ym, label = 'model')
ax2.set_xlabel('$P(X<x)$')
ax2.set_ylabel('$p$')
ax2.set_ylim((0.90, 1))
ax2.legend()









    Out[3]:





<matplotlib.legend.Legend at 0x15d649588>

Model does not assign 0 scores, like the annotators. Model score distribution is skewed left for x > 0.2.

Q: How do differences in distribution over attack probabilities between annotators and models impact estomates of prevalence of personal attacks?



In [4]:

    
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure(figsize=(12, 6))

xa = np.sort(d['annotated']['recipient_score'])
ya = 100 * (1 - (1. * np.arange(len(xa)) / (len(xa) - 1)))

xm = np.sort(d['annotated']['pred_recipient_score'])
ym = 100 * (1 - (1. * np.arange(len(xm)) / (len(xm) - 1)))

ax1 = fig.add_subplot(121)
ax1.plot(xa, ya, label = 'annotators')
ax1.plot(xm, ym, label = 'model')
ax1.set_xlabel('$t$')
ax1.set_ylabel('Percent of comments that are attacks')
ax1.legend()


ax2 = fig.add_subplot(122)
ax2.plot(xa, ya, label = 'annotators')
ax2.plot(xm, ym, label = 'model')
ax2.set_xlabel('$t$')
ax2.set_ylabel('Percent of comments above threshold t')
ax2.set_ylim((0.0, 3))
ax2.set_xlim((0.2, 1.0))
ax2.legend()
plt.savefig('../../paper/figs/model_vs_annotator_percent_attack_distributions.png')

Q: What do comments in different score intervals look like?



In [5]:

    
def check_range(d, col, min, max, n = 10, ):
    df = d.query('%s >= %f and %s <= %f' % (col, min, col, max))
    for i, r in df.sample(n).iterrows():
        print(r[col])
        print(r.clean_diff)
        print('\n')



In [7]:

    
check_range(d['sample'], 'pred_aggression_score',0.6, 0.7)









    



0.6141905188560486
I put my ass and balls into that Trivia section and for you to say its bad is really sad with me. so very wrong!


0.6059567928314209
No actually its a perfectly necessary and relevant little tidbit when talking about a total fucking nutter like Lerner. 


0.6106816530227661
First of all, I was merely going by the definition in anti-semitism.  Secondly, your definition is untenable.  If Hitler decided to wipe out all the Jews because they didn't have blonde hair, that would still be anti-semitism.  If he decided to wipe out the Jews because Jew started with a J, that too would still be anti-semitism.

Finally, it was a fucking joke.  Get over it.  And get over yourself.   06:25, 7 Jan 2004 (UTC)


0.627877950668335
Let's see, first you removed the images stating they're irrelevant and now you're adding them again? Stupid proud Mongol.  


0.6469118595123291
 Shame on you
Feel very ashamed of what you're doing to that Rex G guy. You are an admin unworthy, on the Dutch wikipedia you'd be removed as one thats for sure. 


0.6450621485710144
You are a sockpuppet of a banned user.


0.6029829382896423
"

""and like to raise hell!"" THANK YOU! Thank you for confirming the accuracy of the article; only an idiot would claim that the Hells Angels are anything other than a bunch of murdering, raping, drug-dealing criminal thugs.  Spock, 1 March 2006 1600 (UCT)

"


0.6544660925865173
"

Long time no see, all of you. Well, first..wtf????? Blood what have you smoke? Devil Worshipers? hahahahahahaha that's the most mmm..i don't know...the most crazy thing i've ever heard about ev... Now, Lithium - ilithum.....yeah, cloud nine...nine minus four plus one...six...six six six...they are satanics! and sweet sacrifice...yeah! i'm gonna cut my veins and offer my blood to the evil! yeah! and good enough! the most evil song ever! na-ah...well i know the discussion is over but i wanted to comment about this. About the RoyalGothic comment...evanesence (it's evanescence)''influences depression. Mmm...i don't agree. Another thing, if you want more information about your satanic doubts, DON'T go to EvThreads...they're gonna cyber-eat you alive if you dare to say something like that with that ""theories"". Cya! talk • Ev "


0.6656092405319214
 Arab demon 

I will send you down to hell.


0.6101257801055908
You serious about that shit? Use whatever excuses you want, all you're trying to do is re-establish the entrenched negative bias.

Q: How does the distribution over aggression scores compare across annotators vs models and different data sets

Plot kernel density estimate of aggression score distribution. This is mainly a sanity check that the aggression model is working new data



In [7]:

    
# annotations
sns.distplot(d['annotated']['aggression_score'].dropna(), hist=False, label = 'annotator scores')

# model on annotated data
sns.distplot(d['annotated']['pred_aggression_score'].dropna(), hist=False, label = 'model on annoted data')

#model no admin sample
sns.distplot(d['sample']['pred_aggression_score'].dropna(), hist=False, label = 'model sample')

plt.xlim(-1, 1)
plt.legend()









    Out[7]:





<matplotlib.legend.Legend at 0x1776b2cf8>

Q: How does the distribution over attack probabilities compare across annotators vs models and different data sets

This is mainly a sanity check that the attack model is working new data



In [8]:

    
sns.distplot(d['annotated']['recipient_score'].dropna(), kde =False, norm_hist = True)









    Out[8]:





<matplotlib.axes._subplots.AxesSubplot at 0x14321fef0>



In [9]:

    
sns.distplot(d['annotated']['pred_recipient_score'].dropna(), kde =False, norm_hist = True)
sns.distplot(d['sample']['pred_recipient_score'].dropna(), kde =False, norm_hist = True)









    Out[9]:





<matplotlib.axes._subplots.AxesSubplot at 0x1435d44e0>



In [ ]: